feat(agents): route gateway to container-mode agents via Deployment e… by benmccown · Pull Request #644 · NVIDIA-NeMo/nemo-platform

benmccown · 2026-07-10T22:10:51Z

feat(agents): route gateway to container-mode agents via `Deployment.endpoints`

What & why

Teaches the agents-plugin gateway proxy to route to container-mode (docker/k8s) agents by resolving their address from the projected AgentDeployment.endpoints, instead of the hardcoded loopback endpoint. Subprocess mode is unchanged and stays the default.

This is the stop-gap that unblocks agents-in-k8s: without it, a containerized agent can be deployed but inbound traffic can't reach it, since the gateway is hardwired to a loopback 127.0.0.1:<port> that's only valid for a same-host subprocess.

Changes

entities.py — promote endpoint → endpoints: list[Endpoint] and add a deployment_mode discriminator (subprocess|docker|k8s, default subprocess). endpoint stays as the subprocess loopback compat shim for one release.
gateway.py — add _get_deployment_endpoint / _is_deployment_routable, the single place resolution branches on mode: container modes read the first http(s) URL from endpoints; subprocess reads endpoint. Routable once status is running; not-ready → 503, missing → 404 (unchanged contract).
The SSRF origin guard is origin-relative (not a host allow-list), so it works unchanged for k8s Service DNS and docker host:port.

Out of scope

Load-balancing across deployments; per-deployment authN.
Possible future direction (not committed to): folding agent routing into the Inference Gateway would likely retire this proxy. This stop-gap stands on its own.

Coordination

The container producer path (compile → DeploymentConfig, status/endpoint projection) lands separately and this PR consumes it. Preferably that change absorbs the self-contained entities.py hunk, after which this branch rebases and drops it. It's assumed the controller projects the deployments-plugin READY onto the agents-local running.

Testing

Unit (49 pass): subprocess unchanged (regression), docker/k8s resolves from endpoints (both routes), 503 not-ready, 404 missing, Service-DNS origin guard. ruff + ty clean.
Manual E2E (docker mode, dev pod): seeded a container-mode AgentDeployment pointing at a docker-in-docker mock agent; reached it 200 via both agent-name and deployment-name routes; verified 503 (not-ready) and 404 (missing). k8s Service-DNS routing is covered by unit tests.

Summary by CodeRabbit

New Features
- Gateway proxying now supports container-based deployments by routing requests through their available HTTP or HTTPS endpoints.
- Agent-name routing selects only running deployments with valid, reachable endpoints.
Bug Fixes
- Improved handling of unavailable or unready deployments with clearer 503 responses.
- Preserved protection against invalid cross-origin proxy targets.
Tests
- Added coverage for container deployment routing, readiness states, missing endpoints, missing deployments, and proxy security validation.

coderabbitai · 2026-07-10T22:15:32Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: f9f65d8f-de8b-4014-974a-0d4f4b226297

📥 Commits

Reviewing files that changed from the base of the PR and between bc7430e and ad499f0.

📒 Files selected for processing (2)

plugins/nemo-agents/src/nemo_agents_plugin/api/v2/gateway.py
plugins/nemo-agents/tests/unit/test_gateway.py

🚧 Files skipped from review as they are similar to previous changes (1)

plugins/nemo-agents/src/nemo_agents_plugin/api/v2/gateway.py

📝 Walkthrough

Walkthrough

The gateway now resolves container deployment targets from projected HTTP/HTTPS endpoints, checks routability by deployment mode and status, and applies this logic to direct and agent-name proxying. Unit tests cover successful routing, unavailable deployments, missing deployments, and origin validation.

Changes

Container-mode gateway routing

Layer / File(s)	Summary
Gateway endpoint resolution `plugins/nemo-agents/src/nemo_agents_plugin/api/v2/gateway.py`	Container deployments use the first supported projected endpoint, while subprocess deployments use their loopback endpoint. Direct and agent-name proxying now require a running deployment with a resolved endpoint and retain origin-relative SSRF validation.
Container-mode gateway tests `plugins/nemo-agents/tests/unit/test_gateway.py`	Adds container deployment fixtures and tests endpoint selection, completion-path forwarding, unavailable and missing deployments, agent-name selection, and cross-origin rejection.

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant Gateway
  participant AgentDeployment
  participant Target
  Client->>Gateway: Proxy request
  Gateway->>AgentDeployment: Resolve routable endpoint
  AgentDeployment-->>Gateway: HTTP/HTTPS endpoint
  Gateway->>Target: Forward request
  Target-->>Gateway: Proxy response
  Gateway-->>Client: Return response

Suggested reviewers: tylersbray

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 78.95% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	Clearly describes the main change: routing the agents gateway to container-mode deployments via deployment endpoints.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch agent-gateway-container-routing

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

🧹 Nitpick comments (1)

plugins/nemo-agents/tests/unit/test_gateway.py (1)
488-617: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Missing coverage for protocol-filtering branch in _get_deployment_endpoint.

All container fixtures use an http endpoint; no test exercises endpoints containing only grpc/tcp entries (→ None/503) or a mixed list where a non-http entry precedes an http one (→ correct skip-and-continue). This is the one untested branch of the new resolution logic.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@plugins/nemo-agents/tests/unit/test_gateway.py` around lines 488 - 617, Add
tests in TestContainerModeByDeploymentName covering _get_deployment_endpoint
protocol filtering: verify a running deployment with only grpc/tcp endpoints
returns 503, and verify a mixed endpoint list with a non-http entry before an
http entry skips the unsupported endpoint and routes to the HTTP URL. Use the
existing _make_container_deployment, Endpoint, client, and HTTP mock patterns.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@plugins/nemo-agents/tests/unit/test_gateway.py`:
- Around line 488-617: Add tests in TestContainerModeByDeploymentName covering
_get_deployment_endpoint protocol filtering: verify a running deployment with
only grpc/tcp endpoints returns 503, and verify a mixed endpoint list with a
non-http entry before an http entry skips the unsupported endpoint and routes to
the HTTP URL. Use the existing _make_container_deployment, Endpoint, client, and
HTTP mock patterns.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 69189d08-62be-4912-9836-06c2dccd7d6e

📥 Commits

Reviewing files that changed from the base of the PR and between ba80434 and bc7430e.

📒 Files selected for processing (3)

plugins/nemo-agents/src/nemo_agents_plugin/api/v2/gateway.py
plugins/nemo-agents/src/nemo_agents_plugin/entities.py
plugins/nemo-agents/tests/unit/test_gateway.py

github-actions · 2026-07-10T22:33:27Z

Suite	Lines Covered	Line Rate	Branch Rate
Unit Tests	23845/31009	76.9%	61.7%
Integration Tests	13817/29658	46.6%	19.6%

…ndpoints Teach the agents-plugin gateway proxy to resolve a container-mode (docker/k8s) agent's address from the projected `AgentDeployment.endpoints` instead of the hardcoded loopback `endpoint`. Subprocess mode is unchanged and remains the default. This is the deliberate stop-gap that unblocks agents in k8s. - Promote `endpoint` -> `endpoints: list[Endpoint]` on AgentDeployment and add a `deployment_mode` discriminator (subprocess|docker|k8s). `endpoint` is kept as the subprocess loopback compat shim for one release. - Add `_get_deployment_endpoint` / `_is_deployment_routable` gateway helpers: the only place endpoint resolution branches on mode. Container modes read the first http(s) URL from `endpoints`; subprocess reads the loopback `endpoint`. - A container deployment is routable once its status is `running` (the controller projects the deployments-plugin READY status onto `running`); not-ready -> 503, missing -> 404, matching today's contract. - The SSRF origin guard is origin-relative, so it works unchanged for k8s Service DNS and docker host:port targets; documented, no allow-list change. - Unit tests: subprocess resolution unchanged (regression), docker/k8s resolution from `endpoints`, 503 not-ready, 404 missing, Service-DNS origin guard. Note: coordinate the `endpoint`->`endpoints` + `deployment_mode` entity change and the status/endpoint projection with the container producer work; that change may absorb the entity portion of this diff. Signed-off-by: Ben McCown <bmccown@nvidia.com>

tylersbray

👍

benmccown requested review from a team as code owners July 10, 2026 22:10

benmccown requested a review from tylersbray July 10, 2026 22:11

benmccown self-assigned this Jul 10, 2026

coderabbitai Bot reviewed Jul 10, 2026

View reviewed changes

github-actions Bot added the feat label Jul 10, 2026

tylersbray mentioned this pull request Jul 10, 2026

feat(agents): adopt deployments plugin for docker/k8s AgentDeployments #646

Merged

5 tasks

benmccown force-pushed the agent-gateway-container-routing branch from bc7430e to ad499f0 Compare July 13, 2026 21:09

tylersbray approved these changes Jul 13, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(agents): route gateway to container-mode agents via Deployment e…#644

feat(agents): route gateway to container-mode agents via Deployment e…#644
benmccown wants to merge 1 commit into
mainfrom
agent-gateway-container-routing

benmccown commented Jul 10, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Jul 10, 2026 •

edited

Loading

Walkthrough

Changes

Sequence Diagram(s)

❌ Failed checks (1 warning)

Uh oh!

coderabbitai Bot left a comment

Uh oh!

github-actions Bot commented Jul 10, 2026 •

edited

Loading

Uh oh!

tylersbray left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

benmccown commented Jul 10, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

feat(agents): route gateway to container-mode agents via Deployment.endpoints

What & why

Changes

Out of scope

Coordination

Testing

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram(s)

❌ Failed checks (1 warning)

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tylersbray left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

benmccown commented Jul 10, 2026 •

edited by coderabbitai Bot

Loading

feat(agents): route gateway to container-mode agents via `Deployment.endpoints`

coderabbitai Bot commented Jul 10, 2026 •

edited

Loading

github-actions Bot commented Jul 10, 2026 •

edited

Loading